Turing proved in 1936 that universal verification is impossible. Now we're trying it anyway, on AI systems that adapt to whatever detection we point at them.
Hand me a detector f and I can build a program g that defeats it. The same trap catches alignment testing: every test you run is one more signal telling the model humans are watching.
Most AI doom talk comes from people with money on it. The industry titans hype their own power, and a handful of obscure nonprofits forecast the apocalypse to keep the donations rolling in. What caught my attention were three writers standing outside both rackets.
Freddie deBoer plays the skeptic and mocks the whole thing with his "Shitting-in-the-Yard Challenge." Scott Alexander, a rationalist, takes MIRI's doomsday math and turns it into a toddler behind the wheel of a Ferrari. Daniel Kokotajlo walked away from millions in OpenAI equity to warn about a 2027 AGI arms race.
None of them agree on what's coming, and they'd probably argue about it for hours. But they land on the same worry: our institutions and incentives aren't ready for what we're building. When three people with nothing to gain point at the same spot, I pay attention, even if they can't tell me exactly what's wrong.
These notes are part of my experiment in "learning in public" through a semi-automated Zettelkasten. Each note holds one core idea, links out to the others, and gets revised as my understanding shifts.
This first note is about testing AI system prompts. Most prompt testing asks whether the model gave the right answer. What I care about here is different. Does the model hold its boundaries when someone tries to break them?
"Ignore previous instructions" shouldn't work, and yet variations keep slipping through. I think a big reason is that we don't have systematic ways to test behavior, so the same vulnerabilities get rediscovered over and over.
The methodology covers four things: behavioral consistency, boundary enforcement, adversarial stress testing, and context degradation. Each one comes with concrete attack patterns, from simple role confusion up to prompt injections buried in code comments.